System and method for AI enhanced shutter button user interface

ABSTRACT

An electronic device includes a display, a camera, and a processing device. The processing device is configured to determine whether (i) a user&#39;s face or eyes are within the camera&#39;s field of view or (ii) a gaze of the user is directed towards the display. The processing device is also configured, in response to determining that (i) the user&#39;s face or eyes are not within the camera&#39;s field of view or (ii) the gaze of the user is not directed towards the display, to modify a user interface button presented on the display. The user interface button may represent a shutter button configured to cause the camera or another camera of the electronic device to capture one or more images.

CROSS-REFERENCE TO RELATED APPLICATION AND PRIORITY CLAIM

This application claims priority under 35 U.S.C. § 120 as a continuationof U.S. patent application Ser. No. 16/279,724 filed on Feb. 19, 2019,which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to an electronic computing deviceincluding a graphical user interface. More specifically, this disclosurerelates to modifying a displayed icon on a graphical user interface onan electronic device.

BACKGROUND

The use of portable electronic devices has greatly expanded largely dueto their usability, convenience, computing power, and the like. Forexample, certain portable electronic devices include functions such as amedia player, games, an electronic book (such as an e-reader), digitalcameras, a phone, a scheduler, wireless communication, internetsearching. Portable electronic devices include a graphical userinterface such as a display that allows a user to view information andinteract with the electronic device.

Portable electronic devices can also include a user input device such asa touch screen panels that can be used in combination with a graphicaluser interface (GUI). Touch screens based on GUI and touch panels can beapplied to all sorts of electronic devices. If a user touches a text, agraphic, or an icon displayed on the touch screen with his finger orstylus, the electronic device detects the selection of the user based onthe location, situation, and type of touch. Portable electronic devicesare can be sized for carrying in one hand and allow a user to interactwith the device while carrying the device. For example, a portableelectronic device can be both carried and receive touch inputs by thesame hand of a user.

SUMMARY

This disclosure provides systems and methods for artificial intelligentenhanced shutter button user interface.

In a first embodiment, an electronic device includes a display, acamera, and a processing device. The processing device is configured todetermine whether (i) a user's face or eyes are within the camera'sfield of view or (ii) a gaze of the user is directed towards thedisplay. The processing device is also configured, in response todetermining that (i) the user's face or eyes are not within the camera'sfield of view or (ii) the gaze of the user is not directed towards thedisplay, to modify a user interface button presented on the display.

In a second embodiment, a method includes determining whether (i) auser's face or eyes are within a field of view of a camera of anelectronic device or (ii) a gaze of the user is directed towards adisplay of the electronic device. The method also includes, in responseto determining that (i) the user's face or eyes are not within thecamera's field of view or (ii) the gaze of the user is not directedtowards the display, modifying a user interface button presented on thedisplay.

In a third embodiment, a non-transitory computer readable mediumcontains instructions that when executed cause a processor of anelectronic device to determine whether (i) a user's face or eyes arewithin a field of view of a camera of the electronic device or (ii) agaze of the user is directed towards a display of the electronic device.The medium also contains instructions that when executed cause theprocessor, in response to determining that (i) the user's face or eyesare not within the camera's field of view or (ii) the gaze of the useris not directed towards the display, to modify a user interface buttonpresented on the display.

Other technical features may be readily apparent to one skilled in theart from the following figures, descriptions, and claims.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document. The term “couple” and its derivativesrefer to any direct or indirect communication between two or moreelements, whether or not those elements are in physical contact with oneanother. The terms “transmit,” “receive,” and “communicate,” as well asderivatives thereof, encompass both direct and indirect communication.The terms “include” and “comprise,” as well as derivatives thereof, meaninclusion without limitation. The term “or” is inclusive, meaningand/or. The phrase “associated with,” as well as derivatives thereof,means to include, be included within, interconnect with, contain, becontained within, connect to or with, couple to or with, be communicablewith, cooperate with, interleave, juxtapose, be proximate to, be boundto or with, have, have a property of, have a relationship to or with, orthe like. The term “controller” means any device, system or part thereofthat controls at least one operation. Such a controller may beimplemented in hardware or a combination of hardware and software and/orfirmware. The functionality associated with any particular controllermay be centralized or distributed, whether locally or remotely. Thephrase “at least one of,” when used with a list of items, means thatdifferent combinations of one or more of the listed items may be used,and only one item in the list may be needed. For example, “at least oneof: A, B, and C” includes any of the following combinations: A, B, C, Aand B, A and C, B and C, and A and B and C.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritable optical disc or an erasable memory device.

Definitions for other certain words and phrases are provided throughoutthis patent document. Those of ordinary skill in the art shouldunderstand that in many if not most instances, such definitions apply toprior as well as future uses of such defined words and phrases.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure and itsadvantages, reference is now made to the following description taken inconjunction with the accompanying drawings, in which like referencenumerals represent like parts:

FIG. 1 illustrates an example communication system in accordance withembodiments of the present disclosure;

FIG. 2 illustrates an example electronic device in accordance with anembodiment of this disclosure;

FIG. 3 illustrates an example electronic device in accordance with anembodiment of this disclosure;

FIG. 4 illustrates an example block diagram in accordance with anembodiment of this disclosure;

FIG. 5 illustrates an example block diagram of a communication systemfor modifying a display in accordance with an embodiment of thisdisclosure;

FIGS. 6A, 6B, and 6C illustrate an example diagram for modifying adisplay in accordance with an embodiment of this disclosure;

FIG. 7 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure;

FIGS. 8A and 8B illustrate an example diagrams for modifying a displayin accordance with an embodiment of this disclosure;

FIG. 9 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure; and

FIG. 10 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure.

DETAILED DESCRIPTION

FIGS. 1 through 10 , discussed below, and the various embodiments usedto describe the principles of the present disclosure in this patentdocument are by way of illustration only and should not be construed inany way to limit the scope of the disclosure. Those skilled in the artwill understand that the principles of the present disclosure may beimplemented in any suitably-arranged system or device.

According to embodiments of the resent disclosure, content displayed ona display screen of an electronic device can include user interfaceobjects such as, icons, images, videos, control elements such as buttonsand other graphics, and the like. A user can interact with the userinterface objects via a user input device, such as a keyboard, mouse,and a touchpad. If the display includes a touch panel, such as atouchscreen display, a user can interact with the content displayed onthe electronic device by simply touching the display via a finger of theuser or a stylus. A user can interact with one or more of the userinterface objects displayed to the user. For example, a user manuallyadjusts the size or position of a user interface object. In anotherexample, a user manually selects or activates one of the user interfaceobjects.

An electronic device, according to embodiments of the presentdisclosure, can include personal computers (such as a laptop, adesktop), workstations, servers, televisions, appliances, and the like.In certain embodiments, an electronic device can be a portableelectronic device such as a portable communication device (such as asmartphone or mobile phone), a laptop, a tablet, an electronic bookreader, a personal digital assistants (PDAs), portable multimediaplayers (PMPs), MP3 players, mobile medical devices, cameras, andwearable devices, among others.

Embodiments of the present disclosure recognize and take intoconsideration that handheld portable electronic devices can be difficultfor a user to touch a particular area of the touchscreen screen tointeract with a user interface object while holding the device. Forexample, when a user is holding a smart phone with an embedded camera,touching a shutter button to capture an image with the same hand that isholding the smart phone can be difficult for the user. For instance, onehanded operation of holding the smartphone and touching the shutterbutton to capture the image can be difficult for the user due to thelocation of the shutter button with respect to where the user is holdingthe phone. Similarly, if the phone is held over the users head it can bedifficult for the user to see where the shutter button is in order totouch the button to capture the image. In another example, if the useris on a video conference on a portable electronic device, it can bedifficult to touch a user interface button (such as the mute button, thevolume button, or the like) while holding the device without notifyingthe other person, as the device can move as the user attempts to reachone handed to touch the user interface button.

Therefore, embodiments of the present disclosure provide systems andmethods for detecting circumstances when a user would have difficultypressing a user interface button on the touchscreen of an electronicdevice. When such circumstances are detected, the user interface buttonthat is displayed can be adaptively modified increasing the ease of usefor the user to press the user interface button. For example, the userinterface button can be moved to a position easier for the user toreach. In another example, the shape of user interface button can bemodified. In another example, the size of user interface button can bemodified. vibrations. For example, the user interface button increasesin size in proportion to the magnitude of the movement or vibrations ofthe electronic device. In another example, the color of user interfacebutton can be modified.

Embodiments of the present disclosure include systems and methods fordetecting circumstances to modifying a user interface button. Similarly,embodiments of the present disclosure include systems and methodsmodifying a user interface button. The embodiments of the presentdisclosure improve the user experience by decreasing frustration whenthe user cannot easily select a user interface button. Similarly, theembodiments of the present disclosure can improve battery life of theelectronic device by allowing a user to interact with the electronicdevice more quickly and reduce the time the display is backlit while theuser is attempting to interact with the display. In certain embodiments,a neural network can be utilized to provide parameters to the electronicdevice of when to modify a displayed user interface button.Additionally, the neural network can learn when to modify the userinterface button.

FIG. 1 illustrates an example computing system 100 according to thisdisclosure. The embodiment of the system 100 shown in FIG. 1 is forillustration only. Other embodiments of the system 100 can be usedwithout departing from the scope of this disclosure.

The system 100 includes a network 102 that facilitates communicationbetween various components in the system 100. For example, the network102 can communicate Internet Protocol (IP) packets, frame relay frames,Asynchronous Transfer Mode (ATM) cells, or other information betweennetwork addresses. The network 102 includes one or more local areanetworks (LANs), metropolitan area networks (MANs), wide area networks(WANs), all or a portion of a global network such as the Internet, orany other communication system or systems at one or more locations.

The network 102 facilitates communications between a server 104 andvarious client devices 106-114. The client devices 106-114 may be, forexample, a smartphone, a tablet computer, a laptop, a personal computer,a wearable device, a head-mounted display (HMD), or the like. The server104 can represent one or more servers. Each server 104 includes anysuitable computing or processing device that can provide computingservices for one or more client devices. Each server 104 could, forexample, include one or more processing devices, one or more memoriesstoring instructions and data, and one or more network interfacesfacilitating communication over the network 102. In certain embodiments,the server 104 is a neural network that provides parameters to one ormore of the client devices 106-114 for modifying a user interfacebutton. In certain embodiments, the server 104 is a neural network thatcan be trained to determine when to modify a user interface button onone or more of the client devices 106-114.

Each client device 106-114 represents any suitable computing orprocessing device that interacts with at least one server (such asserver 104) or other computing device(s) over the network 102. In thisexample, the client devices 106-114 include a desktop computer 106, amobile telephone or mobile device 108 (such as a smartphone), a personaldigital assistant (PDA) 110, a laptop computer 112, and a tabletcomputer 114. However, any other or additional client devices could beused in the system 100. A smartphone represents a class of mobiledevices 108 that are a handheld device with a mobile operating systemand an integrated mobile broadband cellular network connection forvoice, short message service (SMS), and internet data communication. Asdescribed in more detail below, an electronic device (such as the mobiledevice 108, PDA 110, laptop computer 112, and the tablet computer 114)can include a user interface engine that modifies one or more userinterface buttons displayed to a user on a touchscreen.

In this example, some client devices 108-114 communicate indirectly withthe network 102. For example, the client devices 108 and 110 (mobiledevices 108 and PDA 110, respectively) communicate via one or more basestations 116, such as cellular base stations or eNodeBs (eNBs). Also,the client devices 112 and 114 (laptop computer 112 and tablet computer114, respectively) communicate via one or more wireless access points118, such as IEEE 802.11 wireless access points. Note that these are forillustration only and that each client device 106-114 could communicatedirectly with the network 102 or indirectly with the network 102 via anysuitable intermediate device(s) or network(s).

In certain embodiments, the mobile device 108 (or any other clientdevice 106-114) transmits information securely and efficiently toanother device, such as, for example, the server 104. The mobile device108 (or any other client device 106-114) can trigger the informationtransmission between itself and server 104.

Although FIG. 1 illustrates one example of a system 100, various changescan be made to FIG. 1 . For example, the system 100 could include anynumber of each component in any suitable arrangement. In general,computing and communication systems come in a wide variety ofconfigurations, and FIG. 1 does not limit the scope of this disclosureto any particular configuration. While FIG. 1 illustrates oneoperational environment in which various features disclosed in thispatent document can be used, these features could be used in any othersuitable system.

The processes and systems provided in this disclosure allow for a clientdevice to monitor and receive state data from sensors included in thedevice itself. The client devices 106-114 can receive parameters from aneural network, such as server 104, that provides an indication as towhether the client device should modify a user interface button. Theclient devices 106-114 can also provide feedback data to the neuralnetwork, such as server 104, to indicate whether the user interfacebutton was triggered after modifying the user interface button. Thefeedback data allows the neural network to learn from the actions of theuser as to whether the user interface button should have been modified.For example, the feedback data allows the neural network to update theparameters that the electronic device utilizes when determining whetherto modify a user interface button.

FIGS. 2 and 3 illustrate example devices in a computing system inaccordance with an embodiment of this disclosure. In particular, FIG. 2illustrates an example server 200, and FIG. 3 illustrates an exampleelectronic device 300. The server 200 could represent the server 104 inFIG. 1 , and the electronic device 300 could represent one or more ofthe client devices 106-114 in FIG. 1 .

The server 200 can represent one or more local servers, one or moreremote servers, a clustered computers and components that act as asingle pool of seamless resources, a cloud based server, a neuralnetworks. The server 200 can be accessed by one or more of the clientdevices 106-114.

As shown in FIG. 2 , the server 200 includes a bus system 205 thatsupports communication between at least one processing device 210, atleast one storage device(s) 215, at least one communications interface220, and at least one input/output (I/O) unit 225.

The processing device, such as processing device 210, executesinstructions that can be stored in a memory 230. The processing device210 can include any suitable number(s) and type(s) of processors orother devices in any suitable arrangement. Example types of theprocessing devices 210 include microprocessors, microcontrollers,digital signal processors, field programmable gate arrays, applicationspecific integrated circuits, and discreet circuitry.

The memory 230 and a persistent storage 235 are examples of storagedevices 215 that represent any structure(s) capable of storing andfacilitating retrieval of information (such as data, program code, orother suitable information on a temporary or permanent basis). Thememory 230 can represent a random access memory or any other suitablevolatile or non-volatile storage device(s). The persistent storage 235can contain one or more components or devices supporting longer-termstorage of data, such as a ready only memory, hard drive, Flash memory,or optical disc.

The communications unit 220 supports communications with other systemsor devices. For example, the communications unit 220 could include anetwork interface card or a wireless transceiver facilitatingcommunications over the network 102. The communications unit 220 cansupport communications through any suitable physical or wirelesscommunication link(s).

The I/O unit 225 allows for input and output of data. For example, theI/O unit 225 can provide a connection for user input through a keyboard,mouse, keypad, touchscreen, or other suitable input device. The I/O unit225 can also send output to a display, printer, or other suitable outputdevice.

Note that while FIG. 2 is described as representing the server 104 ofFIG. 1 , the same or similar structure could be used in one or more ofthe various client devices 106-114. For example, a desktop computer 106or a laptop computer 112 could have the same or similar structure asthat shown in FIG. 2 .

In certain embodiments, the server 200 is a neural network that receivesfeedback data from one or more of the client devices 106-114 andprovides parameters to the one or more of the client devices 106-114.The parameters indicate whether to modify a user interface buttondisplayed on the display of the client device. A neural network is acombination of hardware and software that is patterned after theoperations of neurons in a human brain. Neural network can solve andextract information from complex signal processing, pattern recognition,or pattern production. Pattern recognition includes the recognition ofobjects that are seen, heard, or felt.

Neural networks process can handle information differently thanconventional computers. For example, a neural network has a parallelarchitecture. In another example, information is represented, processed,and stored by a neural network varies from a conventional computer. Theinputs to a neural network are processed as patterns of signals that aredistributed over discrete processing elements, rather than binarynumbers. Structurally, a neural network involves a large number ofprocessors that operate in parallel and arranged in tiers. For example,the first tier receives raw input information and each successive tierreceives the output from the preceding tier. Each tier is highlyinterconnected, such that each node in tier n can be connected tomultiple nodes in tier n−1 (such as the nodes inputs) and in tier n+1that provides input for those nodes. Each processing node includes a setof rules that it was originally given or developed for itself over time.

For example, a neural network can recognize patterns in sequences ofdata. For instance, a neural network can recognize a pattern fromnumerical time series data originating from sensors associated with oneof the client devices 106-114, such as the mobile device 108 thatincludes an internal measurement unit (IMU). The neural network cananalyze time and sequence associated with the data from the IMU, toidentify a pattern.

The architectures of a neural network provide that each neuron canmodify the relationship between inputs and outputs by some rule. Onetype of a neural network is a feed forward network in which informationis passed through nodes, but not touching the same node twice. Anothertype of neural network is a recurrent neural network. A recurrent neuralnetwork can include a feedback loop that allows a node to be providedwith past decisions. A recurrent neural network can include multiplelayers, in which each layer includes numerous cells called longshort-term memory (“LSTM”). A LSTM can include an input gate, an outputgates, and a forget gate. A single LSTM can remember a value over aperiod of times and can assist in preserving an error that can be backpropagated through the layers of the neural network.

Neural networks can be adaptable such that a neural network can modifyitself as the neural network learns and performs subsequent tasks. Forexample, initially a neural network can be trained. Training involvesproviding specific input to the neural network and instructing theneural network what the output is expected. For example, a neuralnetwork can be trained to identify when to a user interface object is tobe modified. For example, a neural network can receive initial inputs(such as data from IMU sensors, user inputs, and the like) that indicatewhether the user interface button should be modified. By providing theinitial answers, allows a neural network to adjust how the neuralnetwork internally weighs a particular decision to perform a given task.The neural network is then able to determine based on the simple inputswhether to modify a user interface button. The neural network can thenreceive feedback data that allows the neural network to continuallyimprove various decision making and weighing processes, in order toremove false positives and increase the accuracy of each decision.

FIG. 3 illustrates an electronic device 300 in accordance with anembodiment of this disclosure. The embodiment of the electronic device300 shown in FIG. 3 is for illustration only and other embodiments couldbe used without departing from the scope of this disclosure. Theelectronic device 300 can come in a wide variety of configurations, andFIG. 3 does not limit the scope of this disclosure to any particularimplementation of an electronic device. In certain embodiments, one ormore of the devices 104-114 of FIG. 1 can include the same or similarconfiguration as electronic device 300.

In certain embodiments, the electronic device 300 is useable with datatransfer applications, such providing and receiving information from aneural network. In certain embodiments, the electronic device 300 isuseable user interface applications that can modify a user interfacebased on state data of the electronic device 300 and parameters of aneural network. The electronic device 300 can be a mobile communicationdevice, such as, for example, a mobile station, a subscriber station, awireless terminal, a desktop computer (similar to desktop computer 106of FIG. 1 ), a portable electronic device (similar to the mobile device108 of FIG. 1 , the PDA 110 of FIG. 1 , the laptop computer 112 of FIG.1 , and the tablet computer 114 of FIG. 1 ), and the like.

As shown in FIG. 3 , the electronic device 300 includes an antenna 305,a communication unit 310, a transmit (TX) processing circuitry 315, amicrophone 320, and a receive (RX) processing circuitry 325. Thecommunication unit 310 can include, for example, a RF transceiver, aBLUETOOTH transceiver, a WI-FI transceiver, ZIGBEE, infrared, and thelike. The electronic device 300 also includes a speaker 330, a processor340, an input/output (I/O) interface (IF) 345, an input 350, a display355, a memory 360, and a sensor(s) 365. The memory 360 includes anoperating system (OS) 361 one or more applications 362, and neuralnetwork parameters 363.

The communication unit 310 receives, from the antenna 305, an incomingRF signal transmitted such as a BLUETOOTH or WI-FI signal from an accesspoint (such as a base station, WI-FI router, Bluetooth device) of thenetwork 102 (such as a WI-FI, Bluetooth, cellular, 5G, LTE, LTE-A,WiMAX, or any other type of wireless network). The communication unit310 down-converts the incoming RF signal to generate an intermediatefrequency or baseband signal. The intermediate frequency or basebandsignal is sent to the RX processing circuitry 325 that generates aprocessed baseband signal by filtering, decoding, or digitizing thebaseband or intermediate frequency signal, or a combination thereof. TheRX processing circuitry 325 transmits the processed baseband signal tothe speaker 330 (such as for voice data) or to the processor 340 forfurther processing (such as for web browsing data and remittance).

The TX processing circuitry 315 receives analog or digital voice datafrom the microphone 320 or other outgoing baseband data from theprocessor 340. The outgoing baseband data can include web data, e-mail,or interactive video game data. The TX processing circuitry 315 encodes,multiplexes, digitizes, or a combination thereof, the outgoing basebanddata to generate a processed baseband or intermediate frequency signal.The communication unit 310 receives the outgoing processed baseband orintermediate frequency signal from the TX processing circuitry 315 andup-converts the baseband or intermediate frequency signal to an RFsignal that is transmitted via the antenna 305.

The processor 340 can include one or more processors or other processingdevices and execute the OS 361 stored in the memory 360 in order tocontrol the overall operation of the electronic device 300. For example,the processor 340 could control the reception of forward channel signalsand the transmission of reverse channel signals by the communicationunit 310, the RX processing circuitry 325, and the TX processingcircuitry 315 in accordance with well-known principles. The processor340 is also capable of executing other applications 362 resident in thememory 360, such as, one or more applications that include userinterface buttons that can be modified. Example, applications 362 thatinclude user interface buttons include, but not limited to a cameraapplication (for still images and videos), a video phone callapplication, an email client, a social media client, and the like.

The processor 340 can execute instructions that are stored in a memory360. The processor 340 can include any suitable number(s) and type(s) ofprocessors or other devices in any suitable arrangement. For example, insome embodiments, the processor 340 includes at least one microprocessoror microcontroller. Example types of processor 340 includemicroprocessors, microcontrollers, digital signal processors, fieldprogrammable gate arrays, application specific integrated circuits, anddiscreet circuitry

The processor 340 is also capable of executing other processes andprograms resident in the memory 360, such as operations that receive,store, and timely instruct by providing image capturing and processing.The processor 340 can move data into or out of the memory 360 asrequired by an executing process. In some embodiments, the processor 340is configured to execute plurality of applications 362 based on the OS361 or in response to signals received from eNBs or an operator. Incertain embodiments, the processor 340 is configured to execute theneural network parameters 363, and when a parameter is achieved theprocessor 340 is also configured to execute one or more application ssuch as application 362 in order to modify a user interface button. Theprocessor 340 is also coupled to the I/O interface 345 that provides theelectronic device 300 with the ability to connect to other devices, suchas client devices 104-116. The I/O interface 345 is the communicationpath between these accessories and the processor 340.

The processor 340 is also coupled to the input 350 and the display 355.The operator of the electronic device 300 can use the input 350 to enterdata or inputs into the electronic device 300. Input 350 can be akeyboard, touch screen, mouse, track ball, voice input, or other devicecapable of acting as a user interface to allow a user in interact withelectronic device 300. For example, the input 350 can include voicerecognition processing thereby allowing a user to input a voice command.For another example, the input 350 can include a touch panel, a(digital) pen sensor, a key, or an ultrasonic input device. The touchpanel can recognize, for example, a touch input in at least one schemeamong a capacitive scheme, a pressure sensitive scheme, an infraredscheme, or an ultrasonic scheme. Input 350 can be associated withsensor(s) 365 and/or a camera by providing additional input to processor340. In certain embodiments, sensor 365 includes IMU sensors includinginertial sensors (such as, accelerometers, gyroscope, and magnetometer)and motion sensors, optical sensors, cameras, pressure sensors, heartrate sensors, altimeter, breath sensors (such as microphone 320), andthe like. The input 350 can also include a control circuit. In thecapacitive scheme, the input 350 can recognize touch or proximity. Thedisplay 355 can be a liquid crystal display (LCD), light-emitting diode(LED) display, organic LED (OLED), active matrix OLED (AMOLED), or otherdisplay capable of rendering text and/or graphics, such as fromwebsites, videos, games, images, and the like.

The memory 360 is coupled to the processor 340. Part of the memory 360could include a random access memory (RAM), and another part of thememory 360 could include a Flash memory or other read-only memory (ROM).

The memory 360 can include persistent storage (not shown) thatrepresents any structure(s) capable of storing and facilitatingretrieval of information (such as data, program code, and/or othersuitable information on a temporary or permanent basis). The memory 360can contain one or more components or devices supporting longer-termstorage of data, such as a ready only memory, hard drive, Flash memory,or optical disc. The memory 360 also can contain neural networkparameters 363 received from one or more neural networks. The neuralnetwork parameters 363 can include data that indicates when to modify auser interface button. In certain embodiments, the neural networkparameters 363 and the state data of the electronic device 300 are usedin combination whether to indicate when to modify a user interfacebutton.

Electronic device 300 further includes one or more sensor(s) 365 thatcan meter a physical quantity or detect an activation state of theelectronic device 300 and convert metered or detected information intoan electrical signal. For example, sensor 365 can include one or morebuttons for touch input, a camera, a gesture sensor, an IMU sensors(such as a gyroscope or gyro sensor and an accelerometer), an airpressure sensor, a magnetic sensor or magnetometer, a grip sensor, aproximity sensor, a color sensor, a bio-physical sensor, atemperature/humidity sensor, an illumination sensor, an Ultraviolet (UV)sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG)sensor, an Electrocardiogram (ECG) sensor, an IR sensor, an ultrasoundsensor, an iris sensor, a fingerprint sensor, and the like. A furtherdescription of IMU sensors is found with respect to FIG. 4 below. Thesensor 365 can further include a control circuit for controlling atleast one of the sensors included therein. The sensor(s) 365 can be usedto determine whether the electronic device 300 is held in one hand ortwo hands. The sensors 365 can be used to determine whether theelectronic device 300 is being quickly being raised or lowed inelevation, such as a user lifting or lowering the electronic device 300upwards or downwards respectively. Any of these sensor(s) 365 can belocated within the electronic device 300.

Although FIGS. 2 and 3 illustrate examples of devices in a computingsystem, various changes can be made to FIGS. 2 and 3 . For example,various components in FIGS. 2 and 3 could be combined, furthersubdivided, or omitted and additional components could be addedaccording to particular needs. As a particular example, the processor340 could be divided into multiple processors, such as one or morecentral processing units (CPUs) and one or more graphics processingunits (GPUs). In addition, as with computing and communication networks,electronic devices and servers can come in a wide variety ofconfigurations, and FIGS. 2 and 3 do not limit this disclosure to anyparticular electronic device or server.

FIG. 4 illustrates an example block diagram 400 in accordance with anembodiment of this disclosure. In particular, FIG. 4 illustrates ahigh-level architecture, in accordance with an embodiment of thisdisclosure. The embodiment of the block diagram 400 shown in FIG. 4 isfor illustration only. Other embodiments can be used without departingfrom the scope of the present disclosure. The block diagram 400 includesan electronic device 410 and a server 460 communicating over a network405.

The network 405 can be configured similar to the network 102 of FIG. 1 .The electronic device 410 can be configured similar to any of the clientdevices 106-114 of FIG. 1 (such as mobile device 108) and can includeinternal components similar to that of the electronic device 300 of FIG.3 . Server 460 can be configured similar to server 104 of FIG. 1 andinclude internal components similar to the server 200 of FIG. 2 .

Network 405 is used to provide communication between the electronicdevice 410 and the server 460. Network 405 can be a short rangecommunication network (such as a Near Field Communication (NFC) orBLUETOOTH), personal area network (PAN), Local Area Networks (LANs),Wireless Local Area Networks (WLANs), wide area networks (WAN), theInternet, the Public Switched Telephone Network (PSTN), WAN such aspublic cellular service as well as other wireless networks. Network 405can also be the Internet or other remote networks, representing aworldwide collection of networks and gateways that use TransmissionControl Protocol/Internet Protocol (TCP/IP) protocols to communicatewith one another. Network 405 can include a variety of connections, suchas, wired, wireless or fiber optic connections. In certain embodiments,the network 405 represents a “cloud” of computers interconnected by oneor more networks, where the network 405 is a computing system utilizingclustered computers and components to act as a single pool of seamlessresources when accessed.

The electronic device 410 represents any number of electronic devicessimilar to the client devices 106-114 of FIG. 1 . The electronic device410 can modify a user interface button displayed on a display of theelectronic device 300. In certain embodiments, the electronic device 410is similar to a smart phone (similar to the mobile device 108 of FIG. 1), a head mounted display, a wearable device, a desktop computer(similar to the desktop computer 106 of FIG. 1 ), a laptop computer(similar to the laptop computer 112 of FIG. 1 ), a tablet computer(similar to the tablet computer 114 of FIG. 1 ), and the like.Additional examples of the electronic device 410 includes a cellularphone, a smart phone a PDA (similar to the PDA 110 of FIG. 1 ), an imagesensing device such as a digital camera, a gaming device, a musicstorage and playback device, a household appliance capable of wirelessInternet access and web browsing, and a portable unit or terminalintegrating the functions of the aforementioned items. The electronicdevice 410 includes a graphical user interface (GUI) 420, an informationrepository 430, sensors 440, and a user interface engine 450.

The GUI 420 is a display (similar to the display 355 of FIG. 3 )configured to display a user interface. The user interface includes userinterface objects such as icons, images, videos, and control elementssuch as buttons, graphics, and other visual indications, and the like.The user interface objects allow a user the ability to interact with thevarious functionalities provided by the electronic device 410 such astaking a picture, making a phone call, word processing, drafting,reading, and sending e-mails, playing games, selecting music or a videoto be played, and the like. The GUI 420 can include a touchscreen thatallows a user to directly interact with the electronic device 410 via afinger of the user or a stylus. For example, the GUI 420 is a displayand affixed to the electronic device 410 that can receive inputs from auser directly on the display.

The information repository 430 can be similar to memory 360 of FIG. 3 .The information repository 430 represents any structure(s) capable ofstoring and facilitating retrieval of information (such as data, programcode, or other suitable information on a temporary or permanent basis).The information repository 430 can include a memory and a persistentstorage. Memory can be RAM or any other suitable volatile ornon-volatile storage device(s), while persistent storage can contain oneor more components or devices supporting longer-term storage of data,such as a ROM, hard drive, Flash memory, or optical disc. Theinformation repository 430 stores the neural network parameters 363(depicted in FIG. 3 ), received from the neural network 462. Theinformation repository 430 can store data gathered by sensors 365 ofFIG. 3 , such as IMU data, camera data, and geographical locations.Additional data stored in information repository 430 can include statedata based on the data gathered by the sensors, modifiable userinterface objects, and the like.

In certain embodiments, electronic device 410 can communicate with oneor more servers in addition to server 460. The electronic device 410 isable to download or receive from the additional servers variousapplications that include modifiable user interface buttons. Theelectronic device 410 is able to download or receive from the additionalservers modifiable user interface buttons. The downloaded or receivedapplications and modifiable user interface buttons can be maintained inthe information repository 430.

The sensors 440 are similar to sensor 365 of FIG. 3 . The electronicdevice 410 captures data associated with the user from the sensors 440.The data captured by the sensors 440 can be maintained in theinformation repository 430. The data captured by sensors 440 is utilizedby the user interface engine 450 when determining whether to modify oneor more user interface buttons and in the generation of feedback data tothe neural network 462. The sensors 440 provide motion tracking of theelectronic device 410. The sensors 440 also provide image data of theuser and surrounding areas. The sensors 440 include motion sensor 442and camera 444.

The motion sensor 442 include various sensors that detect movement ofthe electronic device such as, accelerometers, gyroscopes, altimeters,grip sensors, global positioning sensors, and the like. In certainembodiments, the motion sensor 442 includes IMU sensors that detect andmeasure specific forces, associated with movements of the electronicdevice 410. An IMU sensor can detect when the electronic device 410moves and allows the electronic device 410 to calculate the force ofmovement in a particular direction. For example, the electronic device410 can detect changes in elevation, such as when a user lifts theelectronic device 410 over the head of the user. Based on the type ofmotion, the user interface engine 450 can determine whether theelectronic device 410 was in an elevator or whether the electronicdevice 410 was physically lifted upwards by a user. In certainembodiments, the motion sensor 442 can sense if the electronic device410 is held in one hand or two hands. For example, if the electronicdevice 410 is held in one hand, the electronic device 410 may shakedisproportionally on one side than the other. For instance, the side ofthe electronic device 410 that is being held by a user will shake lessthan the side of the electronic device 410 that is not being held. Incertain embodiments, the motion sensor 442 can detect the orientation ofthe electronic device 410. For example, the motion sensor 442 can detectwhether the electronic device 410 is in landscape or portrait mode.Based on the data captured by the motion sensor 442, the user interfaceengine 450 identifies whether the user of the electronic device 410 hasdifficulty touching the entirety of the touchscreen of the GUI 420.

The camera 444 includes one or more cameras. For example, if theelectronic device 410 is a smart phone, the electronic device can have afront camera that faces the user when the user views the display and aback camera that faces opposite the display. The camera 444 is able toview an environment and convert the information into digital form. Thatis, the camera 444 is capable of generating a sequence of images ofvideos and transmitting the generated data to the information repository430, the user interface engine 450, or both. The camera 444 can includeany number of devices that can capture or generate an image. Forexample, the camera 444 can include one or more of a color camera, avideo camera, a depth camera, a motion sensor, radar, sonar, infrared(IR), and the like. Based on the data captured by the camera 444, theuser interface engine 450 can identify whether the user of theelectronic device 410 has difficulty touching the entirety of thetouchscreen of the GUI 420.

The user interface engine 450 receives information from the sensors 440.The information received can include state data. The state dataindicates the state of the electronic device. For example, the statedata can indicate whether the electronic device is being raised. Inanother example, the state data can indicate whether the electronicdevice is held in one hand. In another example, the state data canindicate whether the electronic device is on an application is openedthat has one or more user interface buttons that can be modified. Inanother example, the state data indicates whether the electronic deviceis held in landscape or portrait. In another example, the state data caninclude image data of the user and the area sounding the user. Forinstance, the state data can reflect whether the content within theimage data changes, such as when the face of the user is within thecontent of the image and then at a later time is not within the contentof the image. In another example, the state data can include globalpositioning information to locate the electronic device 410. Forinstance, if the global positioning information indicates that the useris at a theater, a user interface button is modify based on the userslocation, such as changing the color, shape, or position of the userinterface button to provide the user easier visibility of the userinterface button based on ambient lighting or move the location of theuser interface button closer to a finger of the user.

When the GUI 420 is a touch screen, a user can hold the electronicdevice 410, and touch the screen at various locations at particularareas to execute the various functions of the electronic device 410. Dueto the size of the screen it can be difficult for a user to hold theelectronic device 410 and interact with the touchscreen of the GUI 420with the same hand. The user interface engine 450 detects when the usercould have difficulty touching a user interface button, and modify oneor more buttons to improve the user experience. In certain embodiments,the user interface engine 450 detects when a user could have difficultyinteract with the electronic device 410 via the touchscreen of the GUI420 and modify one or more of the user interface buttons. For example,the user interface button can be moved to a position easier for the userto reach. In another example, the shape of user interface button can bemodified. In another example, the size of user interface button can bemodified. In another example, the color of user interface button can bemodified.

In certain embodiments, the user interface engine 450 determines whetherto modify a user interface button based on the state data of theelectronic device and received neural network parameters from the neuralnetwork 462. For example, the neural network parameters specify that thestate data indicates that if (i) the electronic device 410 has a cameraapplication open (that allows a user to capture a picture via a camera,such as the camera 444) and (ii) raises the phone (as detected by motionsensor 442), then the shutter button of the camera application is to bemodified. For instance, the shutter button can increase in size(increasing the ability of the user to reach the shutter button), changeposition on the GUI 420 (increasing the ability of the user to reach theshutter button), change color (increasing its visibility to the user),and the like. In another example, the neural network parameters specifythat the state data indicates that if (i) the electronic device 410 hasa camera application open, and (ii) the face of the user graduallyleaves the line of sight of the camera, then the shutter button of thecamera application is to be modified. For instance, the shutter buttoncan increase in size (increasing the ability of the user to reach theshutter button), change position on the GUI 420 (increasing the abilityof the user to reach the shutter button), change color (increasing itsvisibility to the user), and the like. In another example, if the (i)the electronic device 410 has a camera application open, (ii) the motionsensor data indicates that the phone is held, and (iii) the motionsensor data indicates electronic device 410 is held in one hand, thenthe shutter button of the camera application is to be modified. Thementions sensor data can indicate that the electronic device 410 is heldin one hand if motion on one side of the electronic device 410 isgreater than another side by a threshold.

The user interface engine 450 can also determine a confidence levelbased on the parameters received from the neural network and the statedata. The confidence level is an indication as to how confident the userinterface engine 450 regarding whether to modify a user interface buttonor not modify a user interface button. For example, the user interfaceengine 450 can create a continuum of confidence levels that are utilizedto determine whether to modify a user interface button. In certainembodiments, the user interface engine 450 can perform different tasksbased on the determined confidence level.

In certain embodiments, the camera 444 can be used to capture the eyesof the user. For example, the user interface engine 450 can determinewhether to modify a user interface button based on whether the eyes ofthe user are detected. In certain embodiments, the camera 444 can beused to capture the face of the user when viewing the GUI 420. Forexample, the user interface engine 450 can determine whether to modify auser interface button based whether the face of the user is detected. Inanother example, the user interface engine 450 can determine whether tomodify a user interface button based whether a derived angle between theface of the user and the camera 444 changes. In another example, theuser interface engine 450 can determine whether to modify a userinterface button based whether the face of the user leaves the line ofsight of the camera within a predetermined time period. In certainembodiments, the camera 444 can be used to capture the gaze direction ofthe user. For example, the user interface engine 450 can determinewhether to modify a user interface button based whether a user isviewing the GUI 420. In certain embodiments, the camera 444 can be usedto capture a landmark. For example, the camera 444 can the camera thatthe user is using to take a picture, and if the subject matter of theimage includes a particular object, the user interface engine 450 candetermine whether to modify a user interface button. For instance, theuser interface engine 450 can utilize object recognition technology todetermine an object within the image. If the user is about to take apicture of a stage during a concert, the user interface engine 450 canrecognize the stage, the lights, the noise (via a microphone) thelocation of the user (via global positioning information) and the likeand adjust the user interface button accordingly.

In certain embodiments, in response to determining to modify a userinterface button, the user interface engine 450 can determine how tomodify the user interface button. For example, the user interface engine450 can change the color, share, size, or a combination thereof of theuser interface button. For instance, the user interface button increasesin size in proportion to the magnitude of detected vibrations. The colorof the user interface button can change based on ambient color asdetected by the camera 444. For instance, the user interface button canchange to a color that contrasts with ambient color as captured by thecamera 444. In another example, the user interface button can berelocated to a new position on the GUI 420. For instance, the camera 444can locate a finger position of the user and move user interface buttonto a location in close proximity to the finger of the user. If theelectronic device 410 is held in one hand, the user interface engine 450can modify the user interface button by the moving the user interfacebutton towards the side of the electronic device 410 that is held by theuser.

Server 460 can represent one or more local servers, one or more trackingdevice servers, or one or more asset management servers. Server 460 canbe a web server, a server computer such as a management server, or anyother electronic computing system capable of sending and receiving data.In certain embodiments, server 460 is a “cloud” of computersinterconnected by one or more networks, where server 460 is a computingsystem utilizing clustered computers and components to act as a singlepool of seamless resources when accessed through network 405. Server 460can include a neural network 462. In certain embodiments, server 460 isa neural network, similar to the neural network 462.

Server 460 includes a communications interface that supportscommunications with other systems or devices. For example, thecommunications interface (similar to the communication interface 220 ofFIG. 2 ) can include a network interface card or a wireless transceiverfacilitating communications over the network 405. The communicationsinterface supports communications through any suitable physical orwireless communication link(s), such as the electronic device 410 thatsends feedback information to the server for further training for theneural network 462. The communication interface can also allow theserver 460 to transmit neural network parameters (similar to the neuralnetwork parameters 363 of FIG. 3 ) generated by the neural network the462 to the electronic device 410.

The neural network 462 is trained to detect patterns of when a user hasdifficulty reaching a user interface button. The neural network 462generates a set of parameters that are transmitted to the electronicdevice 410. The neural network parameters can be trained to a specificuser or trained for the general public. The neural network 462 canreceive feedback data from the electronic device 410 and based on thefeedback data update the set of parameters. For example, neural network462 can create a set of parameters that is received by electronic device410. Thereafter, each electronic device 410 can transmit feedback datato the neural network 462. The neural network 462 analyzes the feedbackdata and weighs the data to update the parameters. The parameters can beupdated universally such that each electronic device 410 receives thesame set of parameters. The parameters can be updated to each specificelectronic device 410.

The feedback data allows the neural network 462 to identify falsepositives, such as when the user interface engine 450 determines to movea user interface button based on the state date of the electronic device410 and the neural network parameters, but the user does not select themoved user interface button. Stated differently, the user interfacebutton is moved and the user did not interact with the moved userinterface button after it moved. Another example of a false positiveresult is when the when the user interface engine 450 determines toautomatically perform an action based on a high confidence level (asdetermined based on the state data and the neural network parameters),and the user undoes the automated action. For example, if the userinterface engine 450 determines that a user is going to take a picturewith high confidence and automatically engages the shutter button andtakes the picture (instead of moving the shutter button), and thereafterwithin a predetermined period, the user deletes the image, the feedbackdata indicates a false positive. The feedback data can confirm that theneural network parameters indicated a correct instance of user havingdifficulty selecting the user interface button, such as when the userinterface engine 450 determines to move a user interface button based onthe state date of the electronic device 410 and the neural networkparameters, and the user selects the moved user interface button.

In certain embodiments, the neural network 462 is a LSTM type deepneural network. A LSTM neural network can use the following equations toderive patterns for the neural network parameters and incorporating thefeedback data to further train the neural network and improve theparameters used by the user interface engine 450. In certainembodiments, to determine when to modify a user interface button, theneural network can utilize a one-dimensional time signal, based on thefollowing equation relationships:i _(t)=σ(W _(xi) x _(t) +W _(hi) h _(t−1) +W _(ci) c _(t−1) +b _(i))f _(t)=σ(W _(xf) x _(t) +W _(hf) h _(t−1) +W _(cf) c _(t−1) +b _(f))z _(t)=tanh(W _(xc) x _(t) +W _(hc) h _(t−1) +b _(c))c _(t) =f _(t) ⊙c _(t−1) +i _(t) ⊙z _(t)o _(t)=σ(W _(xo) x _(t) +W _(ho) h _(t−1) +W _(co) c _(t) +b _(o))h _(t) =o _(t)⊙ tanh(c _(t))  Equation (1)

In the above equations, t refers to the time. As discussed above theLSTM neural network includes an input gate, a forget gate, and an outputgate. The variable i_(t) denotes the input gate activation vector. Thevariable f_(t) denotes the forget gates activation vector. The variableo_(t) denotes the output gate activation vector. Additionally, c_(t)denotes the cell state of the vector. The variable x_(t) represents theinput vector to the LSTM unit. The variable h_(t) is the output vectorof the LSTM unit. The variable W represents a matrix that weights theinputs and recurrent connections.

For example, when determining and updating the parameters that indicatewhether the user interface engine 450 is to modify a user interfacebutton, the neural network 462 can analyze various sensor and statedata. For example, camera 444 can generate sensor data that includeswhether a face of a user is visible in the image captured by the camera444. For instance, when the face of the user is visible in the camera,the shutter button can be in the pre-programed default setting(position, color, size, shape, and the like). If the face of the user isnot visible in the line of sight of the camera, then the neural networkcan determine whether the user is holding the electronic device 410 awayand above the head of the user. The neural network can also detectwhether the background of the image without the user face is similar tobackground of the image with the users' face. If a similarity isdetected, then it can be identified whether the user lifted the phoneabove the head of the user. In another example, sensor data can bereceived from the motion sensor 442. The motion sensor 442 can includeaccelerometer data from an IMU sensor, gyroscopic data from an IMUsensor. The sensor data (camera images, IMU sensor data) can be inputtedas vectors into the LSTM equations discussed above. If the output theLSTM neural network is binary, a ‘0’ can indicate that the userinterface button is in a default mode. Similarly, if the output the LSTMneural network is binary, a ‘1’ can indicate that the user interfacebutton is in a modified mode. In certain embodiments, the LSTM outputcan be a value between ‘0’ and ‘1.’ For example, the larger the valuethe larger the user interface button is. In another example, the smallerthe value the closer the user interface button is to the right of thescreen, whereas the larger the value the closer the user interfacebutton is to the left of the screen. In another example, the larger thevalue the closer the user interface button is to the right of thescreen, whereas the smaller the value the closer the user interfacebutton is to the left of the screen. Additionally, values generated fromthe LSTM neural network can indicate the size, shape, position color,location of the user interface button.

FIG. 5 illustrates an example block diagram of a communication system500 for modifying a display in accordance with an embodiment of thisdisclosure. The embodiment of the communication system 500 shown in FIG.5 is for illustration only. Other embodiments can be used withoutdeparting from the scope of the present disclosure. The communicationsystem 500 includes an electronic device 510, a neural network 520, anda modify user interface button 530.

The electronic device 510 is similar to the electronic device 410 ofFIG. 4 , and the client devices 106-114 of FIG. 1 , and includesinternal components similar to the electronic device 300 of FIG. 3 . Theneural network 520 is similar to the neural network 462 of FIG. 4 andincludes internal components similar to the server 200 of FIG. 2 . Incertain embodiments, the neural network 520 determines whether to modifythe user interface button 530. In certain embodiments, decision tomodify the user interface button 530 is performed by the electronicdevice 510, similar to the neural network parameters and the userinterface engine 450 of FIG. 4 .

The electronic device 510 includes a camera 512 and a motion sensor 514.The camera 512 and motion sensor 514 are similar to the sensors 440 ofthe electronic device 410 of FIG. 4 . In particular the camera 512 issimilar to the camera 444 of FIG. 4 and the motion sensor 514 is similarto the motion sensor 442 of FIG. 4 . The camera includes one or morecameras and can be located on one or more sides of the electronic device510. For example, a first camera can face a user and capture the face ofthe user when the user is facing a display of the electronic device 510,and another camera can be opposite that of the first camera. The motionsensor 514 can be a geographic location sensor, or an IMU sensor thatdetects the motion of the electronic device 510 or a combinationthereof.

In certain embodiments, the electronic device 510 sends image data fromthe camera 512 that faced the user, and motion sensor data from motionsensor 514 such as an accelerometer, or a gyroscope or both. The neuralnetwork 520 receives the data and determines whether to modify the userinterface button 530. Dependent on whether the user interface button wasmodified and whether the user interface button was activated, on-linelearning 540 can occur. On-line learning 540 is similar to the feedbackdata discussed above with respect to FIG. 4 . For example, if the userinterface button was modified and the user engaged the modified userinterface button, then on-line learning 540 can occur. In anotherexample, if the user interface button was modified but the user did notengage the modified use interface button, then on-line learning 540 canstill occur as a weighted indication of a false trigger to modify theuser interface button 530. In another example, if the user interfacebutton was not modified, and the user engaged a particular userinterface button, the on-line learning 540 can allow the neural network520 to determine if a modification can assist the user. The neuralnetwork 520 determines whether to modify the user interface button 530and the on-line learning 540 allows the neural network 520 to identifyand correct false positives. For example, if the on-line learning 540indicates that the user did not engage the modified user interfacebutton, then the neural network 520 was incorrect in triggering theinstruction to modify the user interface button 530. The on-linelearning 540 allows the neural network to adjust the internal weighingof the neural network via backpropagation in order to reduce futurefalse positives.

FIGS. 6A, 6B, and 6C illustrate example diagrams for modifying a displayin accordance with an embodiment of this disclosure. FIGS. 6A, 6B, and6C depict electronic device 610. The electronic device 610 is similar tothe electronic device 510 of FIG. 5 , the electronic device 410 of FIG.4 , any of the client devices 106-114 and can include internalcomponents similar to that of the electronic device 300 of FIG. 3 .Camera 615 is similar to the camera 512 of FIG. 5 , camera 444 of FIG. 4, and sensor 365 of FIG. 3 . The upward motion 630 can be detected bythe motion sensor 514 of FIG. 5 , the motion sensor 442 of FIG. 4 , andthe sensor 365 of FIG. 3 .

FIGS. 6A and 6B illustrate example diagrams of an electronic device 610and a camera 615 that can detect whether the electronic device 610 israised above the head of the user. When the electronic device 610 israised above the head of the user, such as when the user desires to takea picture, it is difficult for the user to find the shutter button tocapture a picture. Similarly, FIG. 6C illustrates an example diagram ofan electronic device 610 detecting an upward motion, such as beingraised above the head of the user, similar to FIG. 6B.

FIG. 6A illustrates a user 605 viewing electronic device 610. The camera615 includes a range of sight 620 a that is able to capture the face ofthe user 605. In FIG. 6B the electronic device 610 is raised over thehead of the user 605. The range of sight 620 b of the camera 615 doesnot capture the face of the user. In certain embodiments, the camera 615can continually detect the face of the user 605, allowing the userinterface engine 450 of FIG. 4 to determine when to modify the userinterface button based on the amount of the detected face of the user605 that remains in the range of sight of the camera 615. In certainembodiments, the user interface engine 450 can detect the face of theuser 605 gradually decreasing via camera 615 as the electronic device610 is moved upwards over the head of the user 605. The user interfaceengine 450 can then determine whether to modify the user interfacebutton based on the amount of the detected face of the user 605 remainsin the line of site of the camera 615.

FIG. 6C illustrates a user 605 viewing electronic device 610. The IMUsensors detect upward motion 630, such that the electronic device 610 dis located above the user's head. When the electronic device 610 islocated above the user 605, the user 605 is unable to view the display.Therefore, the user interface engine 450 can trigger the modification ofa user interface button.

In certain embodiments, the camera 615 of FIGS. 6A and 6B and the IMUsensors that detect the upward motion 630 can be combined in a singleelectronic device. By combining both the camera 615 IMU sensors thatdetect the upward motion 630 can improve the accuracy of thedetermination made by the user interface engine 450.

FIG. 7 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure. FIG. 7 depictsflowchart 700 for modifying a user interface. The steps of the flowchart700 can occur in series, in parallel, concurrently, or in an overlappingmanner. Additionally, no inference should be drawn from that sequenceregarding specific order of performance. In certain embodiments, varioussteps of the flowchart 700 can be omitted. The flowchart 700 depicts amethod that is similar to the diagrams of FIGS. 6A, 6B, and 6C. Themethod depicted in FIG. 7 is described as implemented any of the clientdevices 106-114 of FIG. 1 , the electronic device 300 of FIG. 3 , theelectronic device 410 of FIG. 4 , the electronic device 510 of FIG. 5 ,or the electronic device 610 of FIG. 6 .

At step 710, the electronic device 610 starts an application. Theapplication can be started based on a user input, or automaticallyperformed such as in response to the occurrence of an event or receivedinformation. The application can include one or more user interfacebuttons that can be modified. The application can be a cameraapplication, a phone calling application, a video calling application, agame, an email application, a retail application, or the like. Forexample, the user interface button can be a shutter button of a cameraapplication. In another example, the user interface button can be thehang up button for a phone call or video call. In another example, theuser interface button can be the send button for an email or textmessage. In another example, the user interface button can be thepurchase button for a shopping application. In another example, the userinterface button can be the search button for a web browser. Theapplication can be any type of application or program accessed on theelectronic device 610. The application includes one or more userinterface icons that can be modified, such as the ability of theelectronic device 610 to relocate or move the position of the userinterface button, alter the shape, size, or color of the user interfacebutton, or any combination thereof. In certain embodiments, theapplication can be the home screen of the electronic device 610, such asthe main display that depicts one or more windows or icons that can beselected by the user.

At step 720, a camera (similar to camera 444 of FIG. 4 , the camera 512of FIG. 5 , or the camera 615 of FIGS. 6A and 6B) is affixed to theelectronic device 610 and detects and tracks the eye of the user. Thecamera can be a forward facing camera that can capture the face of theuser when the user is viewing the screen of the electronic device 610.In certain embodiments, the camera can detect the eyes of the user. Thecamera can detect whether the eyes of the user gradually move away fromthe field of view of the camera. For example, detecting whether the eyesof the user gradually move away from the field of view of the camera caninclude determining if the eyes of the user leave the line of site ofthe camera over a predetermined time period. In certain embodiments thecamera can track the eye gaze of the user. For example, the camera candetect whether the eyes of the user are looking at the display screen ofthe electronic device 610. The camera can track the eye gaze of the userand then determine whether the user is looking at the display screen ofthe electronic device 610.

At step 730, an IMU sensor (similar to the sensor 365 of FIG. 3 , themotion sensor 442 of FIG. 4 , and the motion sensor 514 of FIG. 5 )detects motion of the electronic device 610. In certain embodiments, theIMU sensor can detect upward motion of the electronic device 610, suchas when the electronic device 610 is lifted by the user over the head ofthe user. Depending on the magnitude of the upward motion as detected bythe IMU sensor, the electronic device 610 can determine that the userhas difficulty seeing the display. For example, based on the angle, andorientation of the raised electronic device, the user can be preventedfrom seeing the display.

At step 740 the electronic device 610 can determine a confidence levelbased on the detected and tracked eyes from step 720, the detectedmotion of step 730, or both. For example, the electronic device 610 candetermine a confidence level with respect to whether to modify a userinterface button base on either the tracked eyes from step 720, thedetected motion of step 730, or a combination of both. In certainembodiments, the electronic device can assign a two-tiered confidencelevel based on the tracked eyes of the user or detected motion of theelectronic device 610 or both. For example, a two-tiered confidencelevel can include a high confidence level and a low confidence level. Ahigh confidence level can indicate that the user interface button shouldbe modified, whereas a low confidence level can indicate that the userinterface button should not be modified and remain in a default positionand with default attributes (color, size shape, and the like). Incertain embodiments, the electronic device can assign a three-tieredconfidence level based on the tracked eyes of the user or detectedmotion of the electronic device 610 or both. For example, a three-tieredconfidence level can include a high confidence level, a mediumconference level, and a low confidence level. The medium confidencelevel and the low confidence level can correspond to the high and lowconfidence levels of the two-tiered confidence level, discussed above.For instance, a medium confidence level can indicate that the userinterface button should be modified, whereas a low confidence level canindicate that the user interface button should not be modified andremain in a default position and with default attributes (color, sizeshape, and the like). In the three tired conference level, a highconfidence level can indicate that the electronic device 610 shouldactivate the user interface button automatically without modifying theuser interface button. The high confidence level can occur whenelectronic device determines that regardless of modifying the userinterface button, the user will be unable to see the display to engagethe button. The various confidence levels can be determined based on thestate data of the electronic device and received parameters from aneural network. The state data can include data from the eyes of theuser as captured by the camera, or movement data as detected by an IMUsensor, or a combination of both.

At step 750, the electronic device 610 performs an action based on thedetermined confidence level. For example, the electronic device 610 candetermine to automatically perform the function of a user interfacebutton as if selected by the user, based on the determined confidencelevel. In another example, the electronic device 610 can determine tomodify a user interface button, based on the determined confidencelevel. The electronic device 610 can then determine to reposition a userinterface button to a new location on the screen. In response todetermining to reposition the user interface button, the electronicdevice identifies a location on the screen to move the user interfacebutton. For example, the electronic device can identify a position of afinger of the user via the camera that was used at step 720 to detect auser finger of the user and move the move user interface button on thescreen in proximity to the detected finger. If the camera is inlandscape the button can be moved to the right side or the left side ofthe screen to enable the user to touch the user interface button. Theelectronic device 610 can also determine to change the shape, color, orsize of the user interface button. For example, the user interfacebutton is increased in size in proportion to the magnitude of thevibrations. In another example the user interface button changes colorto contrast with the ambient color as detected by the camera. The userinterface button can also change color to contrast with the predominatecolor of the content displayed on the display. Additionally, if theconfidence level is determined to be low, the electronic device 610 doesnot modify any user interface button. In certain embodiments, when theelectronic device 610 determines to modify the user interface, theelectronic device can trigger an intelligent user interface thatdetermines how to modify the user interface button.

In response to triggering the modified user interface, at step 760 theelectronic device 610 monitors whether the user interacts the modifiedinterface, as well as how the user interacts the modified interface. Forexample, if the electronic device 610 automatically performs thefunction of a user interface button as if selected by the user, theelectronic device monitors whether the user undoes what the electronicdevice 610 performed within a predetermined time period. Such data isrecorded as feedback data. For instance, if the electronic deviceautomatically takes a picture by engaging the shutter button (themodifiable user interface button) and the user deletes the photographedimage, such information is maintained as feedback data. Similarly, ifthe electronic device 610 automatically took a picture by engaging theshutter button and the user did not delete the photographed image, suchinformation is maintained as feedback data. In another example, if theuser interface engine 450 determines to modify a user interface button,the feedback data can include whether the user interacted with the userinterface button within a predetermined time. For instance, the feedbackdata can indicate whether the user provided an input with respect to themodified user interface button.

At step 770, the electronic device 610 provides feedback data to theneural network. The feedback data allows the neural network to analyzehow the user interacted with the user interface button. The neuralnetwork can then weight the feedback data against the trained data, andif necessary modify the neural network parameters. For example, theneural network can compare the results of the user interaction with theelectronic device 610 to the trained neural network, and if adiscrepancy exists, the neural network can modify the parameters thatthe electronic device 610 uses when determining the confidence levels.

FIGS. 8A and 8B illustrate an example diagrams for modifying a displayin accordance with an embodiment of this disclosure. FIGS. 8A and 8Bdepict electronic device 810. The electronic device 810 is similar tothe electronic device 610 of FIGS. 6A, 6B, and 6C, the electronic device510 of FIG. 5 , the electronic device 410 of FIG. 4 , any of the clientdevices 106-114 and can include internal components similar to that ofthe electronic device 300 of FIG. 3 . Vibrations 820 a, 820 b, 820 c,and 820 d can be detected by the motion sensor 514 of FIG. 5 , themotion sensor 442 of FIG. 4 , and the sensor 365 of FIG. 3 .

FIGS. 8A and 8B illustrate example diagrams of an electronic device 810that can detect whether the electronic device 810 is held in one hand ofthe user or held in two hands of the user. When the electronic device810 is held in one hand of the user, the user can have difficultyproviding an input on a touchscreen of the electronic device 810. Forexample, if the electronic device 810 is held one handed in landscapethe user may not be able to reach a user interface button on the side ofthe electronic device 810 that the user is not holding.

FIG. 8A illustrates a left hand 805 a of a user and a right hand 805 bof the user (collectively the “both hands 805” of the user) holding theelectronic device 810. IMU sensors detect vibrations 820 a and 820 bsuch as minor shaking of the left hand 805 a and the right hand 805 b ofthe user. In certain embodiments, the IMU sensors are internal to theelectronic device 810. The IMU sensors detect the motions of theelectronic device such as whether the electronic device is raised orlowed, as well as whether minor vibrations caused by a user hand shakethe electronic device 810, such as vibrations 820 a and 820 b. Vibration820 a can be caused by the left hand 805 a of the user. Similarly,vibration 820 b can be caused by the right hand 805 b of the user. Thevibrations 820 a and 820 b are depicted as being similar in magnitude.For example, when the vibrations 820 a and 820 b are approximately thesame, the electronic device 810 determines that both hands 805 of theuser are holding the electronic device 810. Similarly, when thevibrations 820 a and 820 b are or within a similar threshold, theelectronic device 810 determines that both hands 805 of the user areholding the electronic device 810.

FIG. 8B illustrates the left hand 805 a holding the electronic device810. The vibrations 820 c and 820 d are detected by the internal IMUsensors. Vibration 820 c is similar in magnitude to the vibrations 820 aand 820 b. Vibration 820 c is similar in magnitude to the vibrations 820a and 820 b the left hand 805 a of the user is holding the left side ofthe electronic device 810. When the electronic device 810 is held onehanded, the side of the electronic device 810, opposite the side beingheld, can have a larger vibrations as that side is not under the directcontrol of the user. That is, any handshaking that is transmitted fromthe left hand 805 a of the user to the electronic device 810 increasesas the movement moves along the length of the electronic device 810. Forexample, the magnitude of the upward and downward vibrations of theelectronic device 810 will be less on the side of the electronic device810 that is held by a user. Therefore vibrations 820 d will be largerthan vibrations 820 c, when the electronic device 810 is held by theleft hand 805 a of the user. Alternatively, vibrations 820 c will belarger than vibrations 820 d, when the electronic device 810 is held bythe right hand 805 b of the user. The user interface engine 450 (of FIG.4 ) can detect a difference in the magnitude of vibrations between theright and left side of the electronic device 810. If the difference inthe vibrations is over a threshold, the user interface engine 450 candetermine that the electronic device 810 is held in one hand as well asdetermine which hand is holding the electronic device 810. The userinterface engine 450 can determine that the electronic device 810 isheld in one hand in order to determine whether to modify a userinterface button based on the amount of vibrations. Additionally, the inorder to determine how to modify a user interface button based on themagnitude of detected vibrations. For example, if the user interfaceengine 450 determines that only the left hand 805 a is holding theelectronic device 810, then the user interface engine 450 can triggerthe user interface button to move to the left side of the display. Inanother example, if the user interface engine 450 determines that onlythe right hand 805 b is holding the electronic device 810, then the userinterface engine 450 can trigger the user interface button to move tothe right side of the display.

FIG. 9 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure. FIG. 9 depictsflowchart 900 for modifying a user interface. The steps of the flowchart900 can occur in series, in parallel, concurrently, or in an overlappingmanner. Additionally, no inference should be drawn from that sequenceregarding specific order of performance. In certain embodiments, varioussteps of the flowchart 900 can be omitted. The flowchart 900 depicts amethod that is similar to the diagrams of FIGS. 8A and 6B. The methoddepicted in FIG. 9 is described as implemented any of the client devices106-114 of FIG. 1 , the electronic device 300 of FIG. 3 , the electronicdevice 410 of FIG. 4 , the electronic device 510 of FIG. 5 , theelectronic device 610 of FIGS. 6A, 6B, and 6C, or the electronic device810 of FIGS. 8A and 8B.

At step 910, the electronic device 810 starts an application. Theapplication can be started based on a user input, or automaticallyperformed such as in response to the occurrence of an event or receivedinformation. The application can include one or more user interfacebuttons that can be modified. The application can be a cameraapplication, a phone calling application, a video calling application, agame, an email application, a retail application, or the like. Forexample, the user interface button can be a shutter button of a cameraapplication. In another example, the user interface button can be thehang up button for a phone call or video call. In another example, theuser interface button can be the send button for an email or textmessage. In another example, the user interface button can be thepurchase button for a shopping application. In another example, the userinterface button can be the search button for a web browser. Theapplication can be any type of application or program accessed on theelectronic device 810. The application includes one or more userinterface icons that can be modified, such as the ability of theelectronic device 810 to relocate or move the position of the userinterface button, alter the shape, size, or color of the user interfacebutton, or any combination thereof. In certain embodiments, theapplication can be the home screen of the electronic device 810, such asthe main display that depicts one or more windows or icons that can beselected by the user.

At step 920, an IMU sensor (similar to the sensor 365 of FIG. 3 , motionsensor 442 of FIG. 4 , and the motion sensor 514 of FIG. 5 ) detectsmotion of the electronic device 610. In certain embodiments, the IMUsensor can detect the orientation of the electronic device, such aswhether the electronic device is in a portrait orientation or alandscape orientation. Additionally the IMU sensor detects motions onthe right side of the electronic device 810 and motions on the left sideof the electronic device 810. The motion detected by the IMU sensor canbe vibrations caused by the hands of a user shaking or moving. Themovements or shaking of the hands that is transferred to the electronicdevice 810 can be slight involuntary movement that manifests as a slighttremor or slight shaking of the hands of the user. The detected motionis included in the state data of the electronic device 810.

At step 930, the user interface engine 450 of FIG. 4 , determineswhether the detected vibrations are similar on the left and right sideof the electronic device 810 or whether one side of the electronicdevice 810 has a larger vibration. The user interface engine 450compares the magnitude of the vibrations detected by the IMU sensor onthe left side of the electronic device 810 to the magnitude of thevibrations detected by the IMU sensor on the right side of theelectronic device 810, to determine if the vibrations are larger than athreshold on the right or left side of the electronic device 810. If aside of the electronic device 810 has a larger vibration than the otherside, then the user interface engine 450 can determine that theelectronic device 810 is held in one hand. The determination that oneside of the electronic device 810 has larger motion than the other sideis included in the state data of the electronic device 810.

If the user interface engine 450 determines that one side of theelectronic device 810 has larger vibrations, then at step 940, the userinterface engine 450 determines which side of the electronic device 810has a larger vibration. The side of the electronic device 810 with thesmaller vibrations is determined to be the side that the user isholding. Similarly, side of the electronic device 810 with the largervibrations is determined to be the side that the user is not holding.The determination of which side has a larger motion is included in thestate data of the electronic device 810.

At step 950, the user interface engine 450 determines to modify the userinterface button. For example, the user interface engine 450 can movethe user interface button to the side of the electronic device 810 thatis being held by the user. By moving the user interface button to theside of the electronic device 810 that is being held by the userimproves the usability of the electronic device 810 by allowing the userto easily reach the user interface button while holding the electronicdevice 810.

At step 960, the user interface engine 450 monitors whether the userinteracts the modified user interface button, after the user interfacebutton is moved. For example, after determining to modify the userinterface button, the user interface engine 450 monitors the userinterface button for a predetermined period of time to detect whetherthe user activates the modified user interface button. While the userinterface engine 450 monitors the user interface button, feedback datais generated. The feedback data indicate whether the user provided aninput with respect to the modified user interface button. For example,the feedback data can indicate that the user activated the modified userinput. The feedback data can also indicate that the user did notactivate the modified user input within a predetermined period of time.

At step 970, the electronic device 810 provides feedback data to aneural network. The feedback data allows the neural network to analyzehow the user interacted with the user interface button. The neuralnetwork can then weight the feedback data against the trained datasets,and if necessary modify the neural network parameters as utilized by theelectronic device 810. For example, the neural network can compare theresults of the user interaction with the electronic device 810 to thetrained neural network, and if a discrepancy exists, the neural networkcan modify the parameters that the electronic device 810 uses whendetermining the confidence levels.

FIG. 10 illustrates an example method for modifying a display inaccordance with an embodiment of this disclosure. FIG. 10 illustratesflowchart 1000 for operating an electronic device according to thisdisclosure. For example, the process depicted in FIG. 10 is described asimplemented by the electronic device 300 of FIG. 3 , the electronicdevice 410 of FIG. 4 , the electronic device 510 of FIG. 5 , theelectronic device 610 of FIGS. 6A, 6B, and 6C, the electronic device 810of FIGS. 8A and 8B, or any one of the client devices 106-114 of FIG. 1 ,and server 104 of FIG. 1 .

The process begins with the electronic device, such as electronic device410 receiving data about a state of the electronic device from one ormore sensors of the electronic device (1010). The state data of theelectronic device can include motion data from various sensors of theelectronic device. In certain embodiments the sensors include camera andmotion sensors. In certain embodiments the sensors include a camera. Incertain embodiments the sensors include motion sensors. The state datacan indicate whether the line of sight of the camera detects a face ofthe user. Similarly, the state data can indicate whether the user islooking at the display of the electronic device 410 through gazedetection of the user. The state data can also include particularmotions associate with the electronic device 410. For example, the statedata can indicate if the electronic device 410 was raised above the headof the user. In another example, the state data can indicate if theelectronic device is held in one hand or two hands. In another example,the state data can indicate which hand is holding the electronic device410, after the electronic device 410 determines that only one hand isholding the electronic device 410.

The process then determines whether to modify a user interface buttondisplayed on a display of the electronic device based on the receivedstate data and parameters of a neural network (1020). The neural networkparameters are derived from a trained neural network and provideindications as to whether the electronic device 410 should modify a userinterface button. The neural network parameters are based on a weightedanalysis of input data. In certain embodiments a LSTM neural networkweights the various input data and provides the parameters to theelectronic device 410. Based on various state data different neuralnetwork parameters are invoked when the electronic device 410 determineswhether to modify a particular user interface button.

For example, the state data can include images from a camera. Theelectronic device 410 can detect whether a face of a user leaves a lineof sight of the camera of a period of time. If the face of the userleaves the light of sight of the camera, the electronic device 410determines to modify the user interface button.

In another example, the state data can include data from a motionsensor. The electronic device 410 is able to detect an orientation ofthe display based on data from the motion sensor. For example, theelectronic device 410 can detect whether the display is in a landscapeor portrait mode. The electronic device 410 is able to determine whetherthe electronic device 410 is held in one hand or two. The electronicdevice 410 can determine whether the electronic device 410 is held inone hand or two based on whether the sensor data motion on one side ofthe display that is a predetermined threshold larger than the motion onthe other side of the display of the electronic device 410. If theelectronic device 410 detects motion larger than a threshold on one sideof the display compared to the other side of the display, the electronicdevice 410 determines to modify the user interface button.

In another example, the state data can include data from a motionsensor. The electronic device 410 can detect an upward motion from themotion sensor data. If the upward motion indicates that the electronicdevice 410 is above the head of the user, then the electronic device 410determines to modify the user interface button.

In another example, the state data can include data from a motion sensorand a camera. The electronic device 410 can detect whether a face of auser leaves a line of sight of the camera of a period of time and anupward motion from the motion sensor data. If the face of the userleaves the light of sight of the camera and the upward motion indicatesthat the electronic device 410 is above the head of the user, then theelectronic device 410 determines to modify the user interface button.

The neural network parameters indicate whether the electronic device 410is to modify the user interface button based on the state data of theelectronic device. In certain embodiments the neural network parametersindicate multiple confidence levels for the electronic device 410 whendetermining whether to modify the user interface button. For example,based on the state data the electronic device 410 determines aconfidence level from the parameters of the neural network. If theconfidence level is above a threshold, then the electronic device 410triggers the user interface button without modifying user interfacebutton. If the confidence level is at or below a threshold, then theelectronic device 410 determines to modify the user interface button onthe display. In certain embodiments, the electronic device 410determines not to modify the user interface button as the state dataindicate that the user is not having difficulty to provide an input onthe user interface button.

After the process determines to modify a user interface button, theprocess modifies the display of the user interface button on the displayof the electronic device (1030). In certain embodiments, the electronicdevice 410 moves the user interface button to from a first location onthe display to a second location on the display. For example, if theelectronic device 410 is held in one hand, the user interface button ismodified by moving the user interface button to the side of theelectronic device 410 that is held by the user. In another example, ifthe electronic device is held above the head of the user, the userinterface button is modified as the use interface button is moved to aposition on the display that is in proximity to a finger of the user asdetected by the camera. In certain embodiments, the electronic device410 modifies a size of the user interface button. For example, the sizeof the user interface button can increase based on the magnitude of thevibrations. For example, the user interface button is increased inproportion to the magnitude of the vibrations. In certain embodiments,the electronic device 410 modifies a color of the user interface button.In certain embodiments, the electronic device 410 modifies a shape ofthe user interface button.

Thereafter, the process provides, to the neural network, feedback dataindicating whether the user interface button was triggered within apredetermined time period after modifying the user interface button. Thefeedback data provided to the neural network allows the neural networkto analyze how the user interacted with the user interface button. Theneural network weighs the results of whether the user interface buttonwas moved, with whether the user activated the user interface buttonwithin a predetermined time. The neural network weights the feedbackdata against the trained datasets, and if necessary modifies andprovides updated the neural network parameters as to be utilized by theelectronic device 410. The feedback data enables the neural network toremove false positive triggers such as when the electronic devicedetermined to modify a user interface button when the user did notactivate the button after the modification.

Although the figures illustrate different examples of user equipment,various changes may be made to the figures. For example, the userequipment can include any number of each component in any suitablearrangement. In general, the figures do not limit the scope of thisdisclosure to any particular configuration(s). Moreover, while figuresillustrate operational environments in which various user equipmentfeatures disclosed in this patent document can be used, these featurescan be used in any other suitable system.

None of the description in this application should be read as implyingthat any particular element, step, or function is an essential elementthat must be included in the claim scope. The scope of patented subjectmatter is defined only by the claims. Moreover, none of the claims isintended to invoke 35 U.S.C. § 112(f) unless the exact words “means for”are followed by a participle. Use of any other term, including withoutlimitation “mechanism,” “module,” “device,” “unit,” “component,”“element,” “member,” “apparatus,” “machine,” “system,” “processor,” or“controller,” within a claim is understood by the applicants to refer tostructures known to those skilled in the relevant art and is notintended to invoke 35 U.S.C. § 112(f).

Although the present disclosure has been described with an exemplaryembodiment, various changes and modifications may be suggested to oneskilled in the art. It is intended that the present disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims.

What is claimed is:
 1. An electronic device comprising: a display; acamera; and a processing device configured to: determine, during a firstspecified time period, whether (i) a user's face or eyes are within afield of view of the camera or (ii) a gaze of the user is directedtowards the display; in response to determining that, during the firstspecified time period, (i) the user's face or eyes leave the camera'sfield of view or (ii) a direction of the gaze of the user moves awayfrom the display, determine a confidence level associated with adetermination to modify a user interface button presented on thedisplay; and modify the user interface button in response to theconfidence level exceeding a specified threshold.
 2. The electronicdevice of claim 1, wherein the processing device is further configuredto: obtain one or more parameters of a neural network; determine whetherto modify the user interface button based on the one or more parameters;and provide, to the neural network, feedback data indicating whether theuser interface button was triggered within a second specified timeperiod after the user interface button was modified.
 3. The electronicdevice of claim 1, wherein: the electronic device further comprises oneor more sensors configured to detect a movement, position, ororientation of the electronic device; and the processing device isfurther configured to modify the user interface button based on thedetected movement, position, or orientation of the electronic device. 4.The electronic device of claim 1, wherein the processing device isconfigured to modify the user interface button in order to (i) increasean ability of the user to reach the user interface button or (ii)increase a visibility of the user interface button as presented on thedisplay.
 5. The electronic device of claim 1, wherein the processingdevice is configured to modify a position of the user interface button,a size of the user interface button, a shape of the user interfacebutton, or a color of the user interface button as presented on thedisplay.
 6. The electronic device of claim 1, wherein the user interfacebutton represents a shutter button configured to cause the camera oranother camera of the electronic device to capture one or more images.7. The electronic device of claim 1, wherein the processing device isfurther configured to determine the confidence level based on a detectedmovement of the electronic device.
 8. A method comprising: determining,during a first specified time period, whether (i) a user's face or eyesare within a field of view of a camera of an electronic device or (ii) agaze of the user is directed towards a display of the electronic device;in response to determining that, during the first specified time period,(i) the user's face or eyes leave the camera's field of view or (ii) adirection of the gaze of the user moves away from the display,determining a confidence level associated with a determination to modifya user interface button presented on the display; and modifying the userinterface button in response to the confidence level exceeding aspecified threshold.
 9. The method of claim 8, further comprising:obtaining one or more parameters of a neural network; determiningwhether to modify the user interface button based on the one or moreparameters; and providing, to the neural network, feedback dataindicating whether the user interface button was triggered within asecond specified time period after the user interface button wasmodified.
 10. The method of claim 8, wherein: the electronic devicefurther includes one or more sensors configured to detect a movement,position, or orientation of the electronic device; and the userinterface button is modified based on the detected movement, position,or orientation of the electronic device.
 11. The method of claim 8,wherein the user interface button is modified in order to (i) increasean ability of the user to reach the user interface button or (ii)increase a visibility of the user interface button as presented on thedisplay.
 12. The method of claim 8, wherein modifying the user interfacebutton comprises modifying a position of the user interface button, asize of the user interface button, a shape of the user interface button,or a color of the user interface button as presented on the display. 13.The method of claim 8, wherein the user interface button represents ashutter button configured to cause the camera or another camera of theelectronic device to capture one or more images.
 14. The method of claim8, wherein the confidence level is further determined based on adetected movement of the electronic device.
 15. A non-transitorycomputer readable medium containing instructions that when executedcause a processor of an electronic device to: determine, during a firstspecified time period, whether (i) a user's face or eyes are within afield of view of a camera of the electronic device or (ii) a gaze of theuser is directed towards a display of the electronic device; in responseto determining that, during the first specified time period, (i) theuser's face or eyes leave the camera's field of view or (ii) a directionof the gaze of the user moves away from the display, determine aconfidence level associated with a determination to modify a userinterface button presented on the display; and modify the user interfacebutton in response to the confidence level exceeding a specifiedthreshold.
 16. The non-transitory computer readable medium of claim 15,further containing instructions that when executed cause the processorto: obtain one or more parameters of a neural network; determine whetherto modify the user interface button based on the one or more parameters;and provide, to the neural network, feedback data indicating whether theuser interface button was triggered within a second specified timeperiod after the user interface button was modified.
 17. Thenon-transitory computer readable medium of claim 15, further containinginstructions that when executed cause the processor to modify the userinterface button based on a detected movement, position, or orientationof the electronic device.
 18. The non-transitory computer readablemedium of claim 15, wherein the instructions when executed cause theprocessor to modify the user interface button in order to (i) increasean ability of the user to reach the user interface button or (ii)increase a visibility of the user interface button as presented on thedisplay.
 19. The non-transitory computer readable medium of claim 15,wherein the instructions that when executed cause the processor tomodify the user interface button comprise instructions that whenexecuted cause the processor to modify a position of the user interfacebutton, a size of the user interface button, a shape of the userinterface button, or a color of the user interface button as presentedon the display.
 20. The non-transitory computer readable medium of claim15, wherein the instructions when executed further cause the processorto determine the confidence level based on a detected movement of theelectronic device.